Use of Multimedia Input in Automated Image Annotation and Content-Based Retrieval

This research explores the interaction of linguistic and photographic information in an integrated text/image database. By utilizing linguistic descriptions of a picture (speech and text input) coordinated with pointing references to the picture, we extract information useful in two aspects: image interpretation and image retrieval. In the image interpretation phase, objects and regions mentioned in the text are identi ed; the annotated image is stored in a database for future use. We incorporate techniques from our previous research on photo understanding using accompanying text: a system, PICTION, which identi es human faces in a newspaper photograph based on the caption. In the image retrieval phase, images matching natural language queries are presented to a user in a ranked order. This phase combines the output of (i) the image interpretation/annotation phase, (ii) statistical text retrieval methods, and (iii) image retrieval methods (e.g., colour indexing). The system allows both point and click querying on a given image as well as intelligent querying across the entire text/image database.